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EDITORIAL 


ARTICLE REVIEW CHECKLIST: A CRITERION 
CHECKLIST FOR REVIEWING RESEARCH ARTICLES 
IN APPLIED PSYCHOLOGY! 


Over the past couple of years, a large group of reviewers and I have 
been developing a comprehensive checklist of criteria for reviewing re- 
search articles. The purpose of this effort was to provide a heuristic de- 
vice of issues to think about when reviewing an article. As such, we hoped 
that the checklist might be a useful tool for reviewers, authors, and stu- 
dents. 

A two-part Delphi-like procedure was followed. In the first part, a 
preliminary checklist of criteria was developed and circulated to a large 
number of reviewers. It contained 93 criteria and was divided into 14 
categories (e.g., literature review, sample, measures, procedures, anal- 
yses, conclusions, etc.). Reviewers were asked to examine the list and 
think about what they look for when reviewing an article, and then to 
modify the criteria on the list or add criteria to it. They suggested 860 
additional items and 429 modifications to the original criteria. They were 
also asked to send in any existing unpublished checklists they might have, 
and several were obtained containing 135 more items. As the criteria 
were edited and condensed, special effort was made to incorporate all 
the content and much of the specific wording of the reviewers’ sugges- 
tions. The resulting checklist had 246 criteria divided into 16 categories. 

In the second part of the study, the revised checklist was again circu- 
lated to the group of reviewers. This time they were asked to rate each 
criterion in terms of the weighting it should receive when reviewing an ar- 
ticle, and they were asked about their background and experience.” The 
information was used to develop the final version of the checklist. First, 
approximately 9% of the criteria were eliminated because they were rel- 
atively unimportant (e.g., received low ratings) or were ambiguous (e.g., 
many ratings left blank). Second, the checklist was simplified by group- 
ing the criteria within each category into clusters of similar criteria. Fi- 
nally, within each cluster, the criteria were listed in a very gross rank or- 
dering of importance based on the ratings. The final checklist contained 
223 criteria, 44 clusters, and 15 categories. 


1 Special thanks to the reviewers who not only provided the ideas and data for this 
checklist, but who also labor tirelessly with little recognition to make the review process 
work. 


2 This information was used to explore differences between reviewers on the weightings 
they assigned to the criteria. These data will be reported elsewhere. 
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The participating reviewers consisted of the editorial board and ad 
hoc reviewers for Personnel Psychology, and the editorial boards of Jour- 
nal of Applied Psychology and Academy of Management Journal. A total 
of 156 reviewers participated in the first part of the study (65% response 
rate), and 227 in the second part (66% response rate). 

Several obvious limitations of this checklist should be noted. First, 
these criteria are self-report and may reflect social desirability and not 
actual reviewer behavior. In that sense, these data are more prescriptive 
than descriptive. Second, the most important caveat is that the checklist 
is not meant to replace reviewer judgment in any way, but only to provide 
a memory aid to remind reviewers of some potentially important criteria 
to consider. By analogy, it is like a preflight checklist for a pilot. It is 
not meant to replace flying skill, but only to remind pilots not to forget 
anything. Furthermore, the article review checklist is not meant to be 
applied in some mechanical fashion. Reviewers should not lose sight 
of the “big picture” when judging an article. Third, these criteria have 
not been validated against any external measures of scientific quality or 
contribution. 

Several possible uses could be made of the checklist. As noted, re- 
viewers might use it as a memory aid in reviewing manuscripts. In this 
regard, it is obviously unrealistic to expect articles to meet all the cri- 
teria, and only some criteria may be applicable to any given article. It 
may be especially useful to new and inexperienced reviewers, but more 
seasoned reviewers might also appreciate the extensive listing of criteria 
in order to reduce the memory demands of conducting reviews in an in- 
creasingly complex science. Authors might also find the checklist useful 
for evaluating planned research studies. In that role, it could be used to 
make improvements in the studies before they are conducted. It could 
also be used to evaluate the articles before submission in order to deter- 
mine whether all the important topics are addressed, and it can be used 
at this stage to help anticipate possible criticisms and bolster the article 
accordingly. Finally, the checklist might be useful for training graduate 
students by helping them learn how to critically evaluate research. 


Michael A. Campion, 
Editor 
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Article Review Checklist 
A. Importance of Topic 


1. Theoretical importance. 
e Is theoretically important. 
e Can take field in new direction or change future research. 
e Justifies claims of importance on valid and clearly stated assump- 
tions. 


2. Practical importance. 
e Is practically important. 
e Links theory and practice in an important way. 


3. Appropriateness. 
e Is appropriate to the journal and readership. 
e Is a new, emerging, or under-researched topic. 
e Is timely in terms of current trends in the field. 


B. Literature Review 


1. Linkage to most important literature. 
e References Key (1.e., highly relevant) previous studies. 
e Considers recent literature. 
e Recognizes all relevant and important areas of literature. 


2. Framing within the literature. 
e Uses literature to develop the arguments (i.e., not just a review). 
e Fits the study into the logical development of the research area. 
e Justifies direction taken by the study. 


3. Thoroughness and accuracy. 

e Demonstrates understanding of the literature. 

e Draws proper inferences from previous studies, without overstat- 
ing, misinterpreting, misapplying, or selectively reporting. 

e Identifies the major issues and themes in the literature that are 
relevant to the article. 

e Reviews literature critically, pointing out limitations, conflicts, 
and ambiguities in a fair manner (i.e., not too harsh or lenient). 

e Organizes literature properly to facilitate review. 

e References properly (e.g., recognizes seminal and definitive 
works, recognizes original research rather than over reliance on 
reviews and textbooks, minimizes nonscholarly citations, etc.). 

e Avoids tangents, marginally relevant citations, exhaustive listings 
of literature if not needed, and excessive self-citations. 

e Integrates multiple literatures when they are used. 
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e Educates unfamiliar readers enough to evaluate the subsequent 
research. 
e Considers wide range of areas of literature. 


C. Conceptual Development 


1. Adequacy of scope and complexity. 

e Uses correct levels/units of analysis (e.g., behavior, person, job, 
group, organization, etc.). 

e Focuses on most critical variables (i.e., those known to be poten- 
tially influential), and explains rationale for inclusion and exclu- 
sion of variables. 

e Specifies relationships among variables clearly (including impor- 
tance, direction, and size), in multivariate terms where needed 
(e.g., addition, interaction, all else equal, etc.), with special clar- 
ity regarding complicated relationships (e.g., form of interac- 
tions, mediation/moderation, causal models, etc.). 

e Has falsifiable hypotheses. 

e Has appropriate depth of conceptual development. 

e States antecedents and consequences of each focal construct 
clearly, and directions of causation, if relevant to research pur- 
pose. 

e Has hypotheses or research questions that are appropriate to 
level of knowledge and state of research on the topic (e.g., ex- 
ploratory vs. specific/complex). 

e Considers all relevant theories, or range of theories, and uses or 
acknowledges competing theories as necessary. 

e Explains processes underlying the constructs. 

e Specifies boundary conditions or limits of the theory or concep- 
tual domain (e.g., in terms of units, context, and time). 

e Does not force a theoretical framework when the study is essen- 
tially exploratory. 


2. Clarity and logical coherence. 

e Defines constructs/variables clearly and differentiates them from 
similar constructs/variables. 

e Uses theory and arguments that are internally consistent. 

e Uses clear and logical conceptual and theoretical development, 
leading from literature review to hypotheses or theses. 

e States purposes, hypotheses, research questions, and intended 
unique contribution clearly. 

e Reaches logical and clear deductions about the theory or con- 
ceptual development. 

e States assumptions clearly and justifies them based on logic or 
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evidence. 
e Explains basic ideas and arguments clearly enough to be grasped 
by those outside the immediate topic area. 


D. Additional Criteria for Literature Reviews and Conceptual Papers 


1. Thoroughness. 

e Uses suitable approaches to analyzing, synthesizing, integrating, 
and evaluating the studies. 

e Summarizes a large and diverse literature, including all the infor- 
mation in the domain of interest. 

e Pulls together diverse findings from literatures that would be un- 
familiar to researchers, yet pertinent to the topic. 

e Defines the domain and rules for including and excluding articles 
clearly and justifiably. 


2. Uniqueness and incremental value. 

e Goes beyond previous reviews in the area. 

e Provides new insight, calls attention to a new problem, suggests 
new solutions, or otherwise adds value to current thinking. 

e Analyzes the literature critically (e.g., methods, findings, contra- 
dictions, etc.) and suggests improvements for future studies. 

e Goes beyond simply applying theory, and instead improves the- 
ory in some manner. 

e Organizes and explains previous findings, including anomalous 
findings and differences across studies. 

e Develops propositions, hypotheses, or questions for future re- 
search (i.e., tries to influence future research in some way). 

e Reframes the problem with, and integrates diverse issues into, a 
single theoretical framework. 

e Has appropriate timing (e.g., sufficient studies accumulated to 
be reviewed, sufficient time since last review, etc.). 


E. Sample and Setting 


1. Appropriateness. 

e Uses a sample (e.g., people, jobs, etc.) that is appropriate for the 
research question and adequately generalizable. 

e Uses a setting (e.g., lab, field, archival, etc.) that is appropriate 
for the research question and adequately generalizable. 

e Uses a context (e.g., situation, job, organization, etc.) that is ap- 
propriate for the research question and adequately generaliz- 
able. | 
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2. Justifications. 


e Uses acceptable sampling strategy (e.g., random, representative, 
convenience, etc.) of people, jobs, or other important units of 
study. 

e Recognizes proper units of analysis (including nesting) and con- 
siders multiple units of analysis if needed. 

e Has adequate statistical power, reports power analyses, and in- 
terprets nonsignificant results accordingly. 

e Justifies the generalizability of student samples when used. 

e Considers timing of the study relevant to events which could in- 
fluence results. 


. Sufficiency of description. 


e Explains sampling strategy clearly enough to determine degree 
to which statistical inferences can be made. 

e Has acceptable return rates and attrition rates (e.g., has adequate 
efforts to increase return rates, addresses the influence of nonre- 
spondents and drop-outs, has no obvious biases, etc.). 

e Has acceptable explanations for loss of sample, differing sample 
sizes, and so forth. 

e Describes population and sampling plan and size clearly, and 
population parameters appear likely to be accurately estimated. 

e Has adequately detailed demographics. Compares to known 
populations, previous studies, and theories if possible. 


F. Measurement 


1. Operationalization. 


e Operationalizes constructs correctly (e.g., consistent with litera- 
ture, theory, or conceptualization). Defines constructs and the- 
ory well enough so this judgment can be made. 

e Justifies all measures based on purpose, theory, or previous 
research, and measures all critical variables. 


_ Reliability. 


e Has adequate types and levels of reliability (e.g., internal consis- 
tency, inter-rater, test-retest, alternative forms, etc.). 

e Avoids inappropriate single-item measures. 

e Considers agreement (i.e., absolute level differences) as well as 
reliability (i.e., covariation) as needed. 


. Validity. 
e Avoids obvious criterion contamination, or assesses contamina- 
tion adequately. 
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e Uses measures that are free from bias (e.g., halo, social desirabil- 
ity, knowledge of predictor, etc.), are nonreactive, are likely to 
be accurate (e.g., asks questions respondents can answer), and 
have adequate range and variation. 

e Avoids obvious criterion deficiency (e.g., samples content do- 
main fully, uses multiple measures, uses proper criterion devel- 
opment procedures, etc.), or assesses deficiency adequately. 

e Presents evidence of construct validity (e.g., convergent and dis- 
criminant validity) as needed. 

e Uses multiple measures and sources if possible. 

e Has adequate independence between measures. 

e Addresses dimensionality of measures properly in development 
or analysis. 

4. Availability. 

e Uses standardized, readily available, and well-researched instru- 
ments, when available. Explains fully when existing, accepted 
measures are not used. 

e Includes new measures or examples of new measures in text or 
appendix, and provides references for measures available else- 
where. 

e Uses existing organization-collected measures (e.g., turnover, ab- 
senteeism, performance, etc.) as needed, and explains and eval- 
uates them fully. 


5. Procedural adequacy. 

e Distinguishes clearly between measuring perceptions and inten- 
tions versus actual behaviors and outcomes. 

e Addresses levels of analysis and issues of aggregation correctly 
(and avoids ecological fallacy). 

e Forms scales correctly (e.g., weighting scheme logical) and de- 
scribes them fully. 

e Uses adequate scaling and anchoring methodology (e.g., Likert, 
Thurstone, behaviorally anchored, etc.). 

e Uses highest level of measurement reasonably possible (i.e., 
nominal, ordinal, interval, or ratio). 


G. Design—Experimental and Quasi-Experimental 


1. Appropriateness. 
e Uses a high quality experimental design considering the con- 
straints of the topic and setting. 
e Examines questions that are amenable to experimental/quasi- 
experimental research. 
e Uses adequate experimental task when needed given the topic, 
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conceptual development, sample, and setting. 


2. Proper controls. 


e Has appropriate control or comparison groups. 

e Uses truly random assignment procedures and explains them 
fully, or presents adequate evidence for the comparability of 
comparison groups. 

e Uses counterbalancing and statistical controls as needed. 


3. Valid manipulations. 


e Operationalizes the construct manipulations or intervention cor- 
rectly given the literature, theory, or conceptualization. 

e Avoids obvious artifacts or biases (e.g., demand effects, experi- 
menter expectancy, reactivity, evaluation apprehension, etc.). 

e Avoids or minimizes confounding of extraneous variables with 
the independent variable manipulation. 

e Has adequately strong manipulations or interventions, and has 
equivalence between conditions when needed. 

e Includes manipulation checks when needed. 

e Has realistic levels of factors in terms of populations and settings 
to which inferences are to be made, including multiple levels if 
possible to understand the form of the effect. 

e Considers and includes important situational and contextual fac- 
tors. 


4. Threat avoidance. 


e Minimizes and addresses threats to internal validity (e.g., history, 
instrumentation, testing, maturation, selection, regression, mor- 
tality, directionality, confounding, etc.). 

e Minimizes and addresses threats to statistical conclusion validity 
(e.g., see Analyses items, plus reliability of treatment implemen- 
tation, random irrelevancies in the experimental setting, hetero- 
geneity of respondents, etc.). | 

e Minimizes and addresses threats to construct validity (e.g., see 
other Design items, plus construct under-representation or con- 
fusion, insufficient definition, mono-operation bias, confound- 
ing constructs with levels of constructs, etc.). 

e Minimizes and addresses threats to external validity (e.g., see 
Sampling items). 

e Makes appropriate trade-offs between types of validity (and be- 
tween rigor and relevance) given the state of the research on the 
topic and the purpose of the study. 

e Explains, in lab studies, how key dimensions of the phenomenon 
or process under investigation can be adequately simulated in an 
artificial environment. 
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H. Design—Nonexperimental and Cross-Sectional 


1. Appropriateness. 


e Uses a high quality nonexperimental design given the constraints 
of the topic and setting. 

e Examines questions that are amenable to cross-sectional or other 
nonexperimental research (e.g., tests differential predictions and 
alternative explanations rather than a generalized null hypothe- 
sis, examines generalizability of previous experimental research, 
examines topics that cannot be examined experimentally, etc.). 

e Has logical implied directions of causation that are theoretically. 
realistic in light of previous findings and theory and are assessed 
with adequate statistical procedures. 

e Avoids common method variance (i.e., mono-method or percept- 
percept bias), or explains why it is not a likely counter explana- 
tion for results. 


2. Threat avoidance. 


e Includes needed control variables. 

e Uses logical timing of measurement, especially regarding longi- 
tudinal designs. 

e Identifies and addresses the influence of restriction of range, un- 
reliability, and other statistical factors on results. 

e Emphasizes degree of proof of causation correctly, and avoids 
hidden causal language. 

e Addresses and assesses multicollinearity when needed. 

e Addresses model misspecification (e.g., missing variables) when 
needed. 

e Assesses nonlinearity when needed. 

e Uses multiple research designs if possible. 


I, Design—Meta-Analysis 


1. Adequacy of sample of studies. 


e Includes all reasonably available relevant studies (both published 
and unpublished) in the domain of interest, and addresses the 
“file drawer” problem. 

e Cumulates a sufficient number of studies to justify a meta- 
analysis, and avoids second-order sampling as a major limitation. 


2. Procedural adequacy. 


e Uses technically correct analytic procedures. 

e Explains and justifies rules for including and excluding studies. 
e Explains and justifies the coding of study variables. 

e Includes a list of the studies examined or makes it available. 
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e Explains and justifies the methods of finding studies. 

e Aggregates adequately similar measures or constructs (i.e., 
variables have similar construct validity). 

e Uses multiple coders, and reports acceptable reliability. 


3. Incremental value. 

e Goes beyond simply summarizing the data, but also contributes 
in some other important manner (e.g., theory, practice, method- 
ology, etc.; see additional criteria for Literature Reviews and 
Contribution). 

e Explores moderators fully. 


J, Design—Qualitative 


1. Procedural adequacy. 

e Defines the problem or questions to be addressed by the data. 

e Executes the methods and techniques properly. 

e Examines questions that are amenable to qualitative research 
(e.g., new topic area, initial stages of research, theory develop- 
ment, alternative methodology, fresh approach to old problem, 
etc.). 

e Uses qualitative methods that are of high quality for the topic, 
setting, and purpose of the study (e.g., observation, interview, 
etc.). 

e Conducts content analyses correctly, and describes them clearly. 

e Describes procedural details fully, such that replication is possi- 
ble. 

e Justifies sampling frame (e.g., persons, observations, time peri- 
ods, etc.) sufficiently for study purposes. 

e Considers advantages and disadvantages of sample and setting. 


2. Appropriateness of conclusions. 

e Develops and defines conceptual categories fully. 

e Relates conclusions to the problem or question, and to the meth- 
ods used. 

e Develops appropriate theory or conceptual model from the data, 
and data supports the emergence of the theory or model. 

e Specifies and explains linkages among concepts or conceptual 
categories. 

e Considers important contextual factors and other explanatory 
conditions. 

e Describes process of change in the phenomenon if needed. 

e (For quantitative studies). Uses some qualitative procedures and 
data as needed to increase accuracy of measurement, support 
causal inferences, or otherwise help interpret the data (e.g., uses 
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subject matter experts, qualitative pilot studies, focus groups, or 
interviews for planning or data interpretation, etc.). 


K. Procedures 


1. Quality. 

e Uses instructions to participants that are unlikely to improperly 
influence results. 

e Uses procedures in lab studies that are involving and have enough 
impact to be realistic. 

e Avoids procedures for data collection in field studies that are so 
intrusive that there is a risk of changing the phenomenon under 
examination or creating Hawthorne effects. 

e Follows ethical standards for the use of human subjects (e.g., 
informed consent, debriefing, etc.). 

e Conducts pilot tests where appropriate. 


2. Adequacy of description. 

e Explains procedures clearly and in adequate detail (enough to 
allow a replication), yet is reasonably succinct. 

e Includes description of selecting or soliciting participants, spe- 
cific instructions to participants, and efforts to ensure standard- 
ization. 

e Describes special conditions clearly which might compromise le- 
gitimacy of the results (e.g., relationship between investigator 
and organization that might reduce objectivity, study originally 
designed for another purpose or part of another study that 
might affect interpretation of results, etc.). 

e Compares procedures with those of other studies when needed. 

e Describes procedural problems and solutions that might be use- 
ful to other investigators. 


L. Data Analysis and Results 


1. Appropriateness of statistics. 

e Uses analyses that are correct for the research questions or hy- 
potheses, research design, and measures. 

e Reports both descriptive and inferential statistics. 

e Uses both univariate and multivariate statistics as needed. 

e Does not overlook simpler or more sophisticated methods that 
are more appropriate. 

e Includes basic statistics needed for future reviews and meta- 
analyses (e.g., means, standard deviations, reliabilities, intercor- 
relations, etc.). 
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2. Warranted assumptions and appropriate error rates. 

e Demonstrates awareness of major assumptions (e.g., level of 
measurement, independence of observations, homoscedasticity, 
fixed effects, etc.), avoids violating major assumptions or 
assesses degree of violation, or uses statistical procedures that 
minimize effect. 

e Uses significance levels that balance Type I and II errors, limits 
the number of levels used (e.g., to two), and applies them consis- 
tently. 

e Controls experiment-wise error rate (e.g., adequate overall 
test or post-hoc procedure). 

e Uses correct data manipulations and transformations. 

e Avoids the apparent selective reporting of data dredging. 

e Avoids or assesses capitalization on chance (e.g., through cross- 
validation or shrinkage formulas), and has an adequate ratio of 
sample to variables. 


3. Completeness. 

e Reports and discusses effect sizes. 

e Reports confidence intervals and significance levels as needed. 

e Does not report redundant or tangential analyses. 

e Reports analyses and statistics unambiguously and consistently, 
especially novel or sophisticated techniques. Gives additional 
explanation and justification as needed, including references. 

e Takes steps to protect the integrity of the data (e.g., quality con- 
trol over collection and inputting), and examines outliers as 
needed. | 

e Conducts obvious supplemental analyses suggested by the study. 

e Uses tables and figures correctly to help clearly communicate 
results. Uses tables and figures to complement, but not repeat, 
text. 

e Describes analyses in a logical sequence (e.g., descriptive statis- 
tics and manipulation checks first, followed by tests of primary 
hypotheses in order, followed by supplemental analyses, etc.). 

e Explores alternative explanations of the findings when possible. 

e Shows consistency across analytic details (e.g., correct degrees of 
freedom, logical interrelationships among statistics, etc.). 


M. Discussion and Conclusions 


1. Explanation of results. 
e Makes correct inferences from research design and data analyses. 
e Links findings back to original hypotheses and purposes of the 
study. 








EDITORIAL 717 





e Does not over-interpret or under-interpret data and results. 

e Does not simply rehash results, but interprets them in a meaning- 
ful manner. 

e Separates objective results description from subjective interpre- 
tation. 

e Summarizes results briefly. 

e Minimizes discussion of tangential topics or issues. 


2. Derivation of implications. 
e Takes proper care in extrapolating from operationalized variables 
back to constructs. 
e Derives specific theoretical implications. 
e Derives specific practical implications. 
e Relates findings to those of other studies. 
e Places results in larger context of relevant issues where necessary. 
e Provides logical and innovative directions for future research. 


3. Description of limitations. 

e Does not overlook or minimize findings contrary to hypotheses. 

e Identifies and addresses alternative explanations for results. 

« Provides a realistic (and adequately self-critical) delineation of 
limitations and weaknesses. 

e Considers both content and methodological explanations of re- 
sults. 

e Identifies known or suspected boundary conditions or limits on 
generalizability. 

e Considers simplest explanations of the results. 

e Explains serendipitous findings as such. 


N. Presentation 


1. Quality of writing. 

e Presents analyses clearly. 

e Uses ideas in a logical and orderly fashion, and links the parts of 
the article together. 

e Writes well and readably (e.g., simple sentences, active voice, 
proper grammar, jargon and acronyms minimized, consistent 
terminology, parallel style, etc.). 

e Is well organized and has correct content in each part of the 
article. 

e Is objective, impartial, and professional. 

e Explains importance of topic explicitly and introduces it early. 

e Is succinct and parsimonious. 

e Writes well-crafted and thorough pieces (e.g., attention to fine 
details and to broad patterns of integration). 
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e Tells an integrated story that is complete (i.e., addresses all obvi- 
ous questions) and flows from beginning to end of the article. 
e Frames writing in as interesting a manner as possible. 


. Conformance with publication guidelines. 


e Has length commensurate with the contribution. 

e Makes title and abstract adequate summaries of main content 
and contributions of the paper. 

e Presents all citations correctly in the reference list. 

e Follows journal style and format requirements. 


O. Contribution 


. Overall contribution. 


e Makes a theoretical contribution (e.g., advances or challenges, 
not just applies, theory). 

e Makes a practical contribution (e.g., derives findings not already 
commonly accepted by practitioners, evaluates a common prac- 
tice, etc.). Includes considerations of utility, organizational ef- 
fectiveness, employee welfare, policy implications, and so forth. 

e Makes a methodological contribution (e.g., evaluates or proposes 
a new instrument, research strategy, analytical technique, etc.). 

e Provides a constructive replication (e.g., replicates but extends in 
an important way). 


. Increment to the current literature. 


e Fills gaps in current knowledge. 

e Goes beyond previous literature in the area. 
e Contributes in nontrivial or nonobvious way. 
e Stimulates potential future research. 


. Creativity and scope. 


e Addresses the “so what?” question. 

e Is innovative and creative. 

e Reports large amount of data and ideas not already reported 
elsewhere (e.g., avoids slicing the data, serial reporting, etc.). 

e Reflects an adequate magnitude or scope of research project. 


. Publication potential. 

e Is likely to improve contribution substantially with revision of 
article. 

e Has strengths in some parts of the study that offset weaknesses 
in other parts. 
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